Comments for MEDB 5502, Week 02

Topics to be covered

What you will learn
- Three models for predicting a continuous outcome
- Analysis of variance table
- Variable selection
- Stepwise regression
- Residual analysis
- Collinearity
- Mediation

Three models

All use a continuous dependent (outcome) variables
All include multiple independent variables
Multiple linear regression (Week 02)
- All independent variables are continuous
Analysis of covariance (Week 03)
- Mix of continuous and categorical independent variables
Multi-factor analysis of variance (Week 04)
- All independent variables are categorical

Why three models?

Historical precedents
Different issues
- Multicollinearity
- Mediator variable
- Risk adjustment
- Moderator variable
- Interactions

If it seems arbitrary to have such different names, you are right. Some of this is historical. If you invent something, you get the right to name it, and different inventors for multiple linear regression, analysis of covariance, and multifactor analysis of variance just decided to use different names.

There are also issues that arise far more often in one model than the other. Multicollinearity, for example, is reserved primarily for multiple linear regression. You’ll hear discussion about multicollinearity this week as well as mediator variables. Risk adjustment is a topic for next week under analysis of covariance, and you will see discussion of moderator variables and interactions two weeks from now under multifactor analysis of variance. In theory, these issues are not exclusive to one model or another. It’s just easier to talk about them within a particular context.

The general linear model

Single model that unites all three models.
Use of indicator variables for categorical data
Not the same as general IZED linear model

It didn’t take long for researchers to discover a common linkage between multiple linear regression, analysis of covariance, and multifactor analysis of variance. In particular, you can treat categorical variables as if they were continuous through the use of indicator variables. You’ll see more discussion about this later.

One bad thing about statistics are all the terms that sound almost the same but mean something quite different. You may have noticed “moderator” and “mediator” from an earlier slide. I can never remember which is which. You’ll also learn about sensitivity and specificity in week 7 (Diagnostic testing) and these are also so easily confused.

But the worst, by far, are the general linear model and the generalized linear model. The IZED added to the end of “general” makes it an entirely different model.

You may see a bit about the generalized linear model in week 7 (logistic regression).

In any case, I have really preferred the use of a single model, the general linear model, in place of multiple linear regression, analysis of covariance, and multifactor analysis of variance. I am a “lumper” and not a “splitter”. For what it is worth, SPSS forces you to use the individual approaches at times instead of the single general linear model.

The dialog box for the general linear model

Break #1

What you have learned
- Three models for predicting a continuous outcome
What’s coming next
- Analysis of variance table

Data for multiple linear regression

Dependent variable: \(Y_1, Y_2,\ldots,Y_n\)
First independent variable: \(X_{11}, X_{12}, \ldots, X_{1n}\)
Second independent variable: \(X_{21}, X_{22}, \ldots, X_{2n}\)
kth independent variable: \(X_{k1}, X_{k2}, \ldots, X_{kn}\)

Matrix notation, 1 of 3

\[\begin{matrix}Y=& \begin{bmatrix} Y_1\\ Y_2\\ \vdots\\ Y_{j}\\ \vdots\\ Y_n \end{bmatrix} \end{matrix}\]

Matrix notation, 2 of 3

\[\begin{matrix}X=& \begin{bmatrix} X_{11} & X_{21} & \ldots & X_{i1} & \ldots & X_{k1}\\ X_{12} & X_{22} & \ldots & X_{i2} & \ldots & X_{k2}\\ \vdots & \vdots & & \vdots & & \vdots\\ X_{1j} & X_{2j} & \ldots & X_{ij} & \ldots & X_{kj}\\ \vdots & \vdots & & \vdots & & \vdots\\ X_{1n} & X_{2n} & \ldots & X_{in} & \ldots & X_{kn} \end{bmatrix} \end{matrix}\]

Matrix notation, 3 of 3

\[\begin{matrix}X=& \begin{bmatrix} 1 & X_{11} & X_{21} & \ldots & X_{i1} & \ldots & X_{k1}\\ 1 & X_{12} & X_{22} & \ldots & X_{i2} & \ldots & X_{k2}\\ \vdots & \vdots & \vdots & & \vdots & & \vdots\\ 1 & X_{1j} & X_{2j} & \ldots & X_{ij} & \ldots & X_{kj}\\ \vdots & \vdots & \vdots & & \vdots & & \vdots\\ 1 & X_{1n} & X_{2n} & \ldots & X_{in} & \ldots & X_{kn} \end{bmatrix} \end{matrix}\]

Why use multiple linear regression

Two competing purposes
- Mechanisms
  - What variables have an impact on your outcome?
- Prediction
  - What outcome will you see on tomorrow’s patient?
  - “It is difficult to make predictions, especially about the future.” Niels Bohr

It is important to note that there are two competing reasons for using multiple linear regression (and other statistical models as well). The first is to try to understand the underlying mechanisms that explain the “how”. You are looking to understand how a disease harms or how a treatment helps. This could be called a “white box” feature that you get with multiple linear regression. The size and signs of the regression coefficients help you understand what factors contribute to the “how”.

You might think of a mechanistic goal as being able to peek under the hood. Some people are perfectly happy driving a car without having any idea of how the engine works. This is fine, and you are treating the automobile as a black box. You don’t know what goes on underneath the hood of the car. Other people want to understand how a car works and they believe (properly in my opinion) that this understanding makes them a better driver.

Alternatively, you can use a multiple linear regression model to predict the future. You want to tell a patient about his or her prognosis, you want to estimate the staffing needs in a hospital, or you want to estimate the cost savings associated changes in health care delivery.

Predicting the future is a perilous process. Pretty much every investment opportunity carries a warning “Past performance is no guarantee of future results

Multiple linear regression model

\(Y_i=\beta_0+\beta_1 X_{1i}+\beta_2 X_{2i}+\ldots+\beta_k X_{ki}+\epsilon_i\)
- Assumptions: \(\epsilon_i\) are
  - independent,
  - same variance,
  - normally distributed.
- Independent variables must be on interval or ratio scale
  - Nominal/ordinal scales require some care

Estimates for the multiple linear regression model

Use least squares to estimate \(\hat\beta_0,\ \hat\beta_1,\ \hat\beta_2,\ \ldots,\ \hat\beta_k\)
- \(\hat Y_i=\hat\beta_0+\hat\beta_1 X_{1i}+\hat\beta_2 X_{2i}+\ldots+\hat\beta_k X_{ki}\)
- \(e_i=Y_i-\hat Y_i\)

Sum of squares

“Bad” regression results.
- All \(\hat Y_i\) close to each other
- Few \(\hat Y_i\) close to \(Y_i\)
“Good” regression results.
- The \(\hat Y_i\) are spread out
- Many \(\hat Y_i\) close to \(Y_i\)

Let’s talk about a bad regression result. The word “bad” is a bit judgmental. You might substitute “disappointing” but that is also troublesome. Maybe “not too useful” is what I really mean.

Let me give a hypothetical example. Suppose you paid a statistician a fortune to produce a linear regression model that predicted the average length of stay for heart attack patients of a particular age with particular systolic and diastolic blood pressure.

An 84 year old patient with 130/90 blood pressure? The model predicts 2.7 days length of stay.

A 35 year old patient with 120/80 blood pressure? The model predicts 2.4 days length of stay.

A 69 year old patient with 190/140 blood pressure? The model predicts 2.6 days length of stay.

You ask yourself, why are we bothering collecting information about age and blood pressure. The predictions are pretty much all the same.

Another way of looking at this is that a “bad” or “not too useful” regression model is one where the predictions are not very close to the actual data values.

In contrast a “good” or “useful” regression model is one where the predictions change a lot for patients with different characteristics. A wide spread of the predicted values means that you have information that can help you. The first patient’s predicted length of stay is 6.8, the second is 1.5 and the third is 3.1? That’s useful information.

Another way of thinking about a “good” regression model is that it’s predictions are generally pretty close to the actual values.

The analysis of variance table helps you identify “good” and “bad” regression models.

Analysis of variance table for multiple linear regression, 1 of 3

SSR or \(SS_{regression}=\Sigma(\hat Y_i-\bar Y)^2\)
SSE or \(SS_{error}\) or \(SS_{residual}=\Sigma(Y_i-\hat Y_i)^2=\Sigma\ e_i^2\)
SST or \(SS_{total}=\Sigma(Y_i-\bar Y)^2\)

Analysis of variance table for multiple linear regression, 2 of 3

\(df_{regression}=k\)
\(df_{error}=n-k-1\)
\(df_{total}=n-1\)
- \(MS=SS/df\)

Analysis of variance table for multiple linear regression, 3 of 3

\(F=MSR/MSE\)
This tests the hypotheses
- \(H_0:\ \beta_1=\beta_2=\ldots=\beta_k=0\)
- \(H_1:\ \beta_j \ne 0\) for at least one j
- Accept \(H_0\) is F is close to 1
- Reject \(H_0\) is F is much larger than 1

Example using fat data

R-squared

\(R^2=SSR/SST\) or \(1-SSE/SST\)
- Proportion of explained variation

Example using the fat data

10,548.480/15,079.017 = 0.70

Adjusted \(R^2\)

\(1-\frac{MSE}{MST}\) or
\(1-\frac{SSE}{SST}\frac{(n-1)}{(n-k)}\)
- Field textbook suggests a more complex formula
- Penalizes for model complexity (but not enough)

Live demo, ANOVA and R-squared

Break #2

What you have learned
- Analysis of variance table
What’s coming next
- Variable selection

Avoid needlessly complex regression models

“Everything should be made as simple as possible, but not simpler” Albert Einstein (?)
“If you have two competing ideas to explain the same phenomenon, you should prefer the simpler one.” Occam’s Razor
“The parsimony principle for a statistical model states that: a simpler model with fewer parameters is favored over more complex models with more parameters, provided the models fit the data similarly well.” - ClubVITA

Choosing independent variables is a balancing act

Too many is bad
- Expensive
- May not replicate well
Too few is bad
- May not explain much of the variation
- May miss important features
Finding the right balance depends on your goals
- Mechanistic approach versus prediction

I have heard many times that Statistics is as much of an art as it is of a science, and I largely agree with this perspective. The selection of independent variables in a multiple linear regression model is an example of the artistic side of statistics.

It is a balancing act. You want a model that is as simple as possible, but not so simple as to oversimplify. This is a very subjective criteria, but it should not be ignored.

Selecting too many independent variables is definitely bad. It can be expensive. Looking at the data on body fat measurements, do you really need to measure circumferences at 10 different parts of the body to get a good handle on how much body fat someone has? Maybe you could do almost as well with only measurements at 3 locations.

The other problem with overly complex models is that they tend to do a poor job of replication. You might do really well with your own dataset, but when others collect similar data, the overly complex models generally don’t look quite as good. You may even find that new data collected at your own location does work so well with a really complex model.

A model that is too simple might not explain enough of the variation in the data. It may also miss out on important features.

Deciding where to go between the simplest models and the most complex models is a tricky choice. It may depend on the goals of your research. If your goal is to try to understand the mechanisms behind a disease or behind a health care delivery process, you might gravitate towards the simpler models. Including too many variables just makes it harder to understand what is going on “under the hood”. Too many variables is less of a problem if your main goal is prediction. You can tolerate a few extra unneeded variables. The complexity doesn’t hurt you as much if you are only interested in prediction.

I’ll return to this topic in the next section.

Counterpoint on complexity

Machine learning algorithms
Risk adjustment

Rule of 15

Developed by Frank Harrell in a different context
- Ratio of observations to independent variables
  - n/k > 15
  - Some use 10 instead of 15
- Smaller ratios imply poor replicability
Not a replacement for a power calculation
Some researchers have argued against this rule

A commonly cited rule about model complexity is the rule of 15. It is a reminder that the more complex your statistical model, the more data you need.

This rule was originally developed by Frank Harrell in a slightly different context, logistic regression. It is commonly applied in a multiple linear regression context, and it seems reasonable enough.

The rule is that the number of observations in your dataset need to be a lot larger than the number of independent variables. Fifteen times larger, as a matter of fact. So if you have 20 independent variables, then you need at least 300 observations to insure a ratio of 15 to 1. If you have 6 independent variables, you need at least 90 observations.

Now there is nothing stopping you from running a regression model with fewer observations. If you have 50 observations and you want to run a model with 10 independent variables (a 5 to 1 ratio), there is nothing to stop you.

The rule of 15 is a rule of thumb and no one ever got thrown in jail for violating a rule of thumb.

The concern with running a really complex model with a small number of observations is that these models do not replicate well.

Now some researchers argue that a 10 to 1 ratio of observations to independent variables is fine.

Other researchers have argued that this rule of thumb is not valid and that it is okay to run complex models on small datasets. This is a minority viewpoint, however, and running a 10 variable model on just 50 observations is likely to get you in trouble.

An important point to remember here is that k is the number of candidate variables that are considered at some time during the data analysis. It is NOT the number of variables in the final model.

Comparing models with k and k-1 predictors

\(Y_i=\beta_0+\beta_1 X_{1i}+\beta_2 X_{2i}+...+\beta_k X_{ki}+\epsilon_i\)
- Test \(H_0:\ \beta_j=0\) versus \(H_0:\ \beta_j \ne 0\)
  - Remaining \(\beta\)’s could be anything
- Use least squares to estimate \(\hat\beta_0,\ \hat\beta_1,\ \hat\beta_2,\ ...,\ \hat\beta_k\)
- Compute \(T=\frac{\hat\beta_j}{se(\hat\beta_j)}\)
- Accept \(H_0\) if T is close to zero
- Reject \(H_0\) if T is large negative or large positive

Testing the impact of chest circumference

Change in R-squared

\(Partial\ R^2=0.700-0.693=0.007\)

The regression model with chest, abdomen, and hip accounts for 70% of the variation, but a model with just abdomen and hip does almost as well, accounting for 69.3% of the variation. The difference, 0.7% is the amount of additional variation accounted for when you add chest to a model that already included abdomen and hip.

That seems to contradict the earlier finding with the large negative test statistic and the small p-value. But actually, what it is saying is that although there is sufficient statistical evidence to conclude that chest circumference has a real impact, that impact is small. You will see this a lot, especially with datasets with very large sample sizes. You can often achieve statistical significance for an individual variable, but the practical impact can still be negligible.

Partial F test

Test \(H_0:\ \beta_{k+1}=\beta_{k+2}=...=\beta_{k+m}=0\)
Calculate ANOVA table for k+m independent variables
- Note various SS and MS values for the “full” model
Recalculate ANOVA table for only k independent variables
- Note various SS and MS values for the “reduced” model.
Calculate \(F=\frac{(SSR_{full}-SSR_{reduced})/m}{SSE_{full}/(n-(k+m+1))}\)

Testing the impact of hips and chest, 1 of 2

Testing the impact of hips and chest, 2 of 2

\(F=\frac{(SSR_{full}-SSR_{reduced})/m}{SSE_{full}/(n-(k+m+1))}\)
\(F=\frac{(10,548.480-9984.086)/2}{4530.537/(252-(1+2+1))}\)
\(F=\frac{282.2}{18.26829}=15.4\)

Live demo, partial F test

Break #3

What you have learned
- Variable selection
What’s coming next
- Stepwise regression

What is a good model

“Everything but the kitchen sink” model
Stepwise models
Hierarchical models
“Maximimum adjusted R-squared” model
“Out of sample error” model
“Use your brain” model
Some combination of the above?

“Everything but the kitchen sink” model

Include anything that seems remotely plausible
Advantages
- Simple computationally
Disadvantages
- Does not identify mechanisms
- Can be expensive
- Fails if k is very large

Kitchen sink model for body fat

Stepwise model, 1 of 2

Forward selection
Backward elimination
Composite of both

A stepwise model uses p-values to identify which variables to add to the model (forward selection) or which variables to remove from the model (backward elimination). You can combine these two approaches.

Forward selection starts with nothing. It looks at the p-values for every single variable model and chooses the variable with the smallest p-value. Then it looks at every possible two variable model that includes the first selected variable and chooses the second variable based on the smallest p-value. Then it looks at every possible third variable in a model that includes the first two variables. And so forth.

Backward elimination starts with everything. It looks at all the p-values for the k indepdent variables in the model and eliminates the variable with the largest p-value. Then it looks at all the p-values in the model with k-1 independent variables and removes the independent variable with the smallest p-value.

Both approaches take a “one step at a time” approach. You might be tempted in the backward elimination step to toss out every independent variable with a large p-value. But the p-values do change as you start removing variables. One of your independent variables might have a large p-value when you have all k variables in the model, but that p-value might shrink as other variables are tossed out.

A general approach is to start with a forward selection approach, but each time a new variable is added, you check to see if one of the variables which had a promising p-value earlier now looks not so hot. So each forward step is followed with a backward elimination check.

Stepwise model, 2 of 2

Advantages
- Automated in SPSS and other packages
Disadvantages
- Does not incorporate medical knowledge
- Does not control Type I error rate

The forward selection and backward elimination approaches are easy to program and are available in SPSS and most other statistical software packages.

Stepwise regression, however, does not incorporate medical knowledge into the process. You might have two variables that both have large p-values in a backward elimination approach. One is well known to be important by anyone with a medical background, but if it’s p-value is a bit larger than the second one that is a bit of a stretch, you will make the wrong choice, at least from a medical perspective.

The other criticism of stepwise models is that they do not control the Type I error rate. You use repeated hypothesis tests and the chances of making a Type I error and including a variable that does not really belong gets inflated. The number of tests that stepwise regression requires is very large, even if you only have a have only a handful of variables.

Hierarchical models

Sometimes variables fall into natural groups.
- Demographic features
- Patient frailty
- Environmental factors
- Current treatments
Enter data in blocks

“Maximimum adjusted R-squared” model

Fit all possible models (\(=2^k\))
- Select model with largest adjusted R-squared
- Alternative criteria
  - Mallows Cp
  - AIC, AICc
  - LASSO

If the number of independent variables is not too big, you can look at every possible combination of independent variables. For the body fat example, there are ten measures and this means examining 1,024 different models. Not easy for you maybe, but your computer can do this in the blink of an eye.

Be careful, though. You may notice that \(2^k\), the number of models, increases (literally) exponentially. With 20 independent variables, you would need to fit over a million models. With 30, it would be over a billion.

You cannot choose the model that maximizes R-squared. R-squared never decreases as the model becomes more complex. It may only increase by a trivial amount, but because it never decreases. Choosing the model with the largest R-squared value will always select the most complex model, the one with every independent variable.

Adjusted R-squared can go down, because it adjusts for the degrees of freedom. In general, though, adjusted R-squared doesn’t do enough adjusting and will tend to select needlessly complex models. There are other criteria: Mallows Cp, Akaike Information Criteria (AIC), a corrected version of AIC (AICc), and LASSO. These all do a better job of avoiding overly complex models.

“Out of sample error” model

Split data into training/test sets
- Use training data to build the model
- Use test dataset to evaluate fit

“Use your brain” model

Previous research
Knowledge of medicine/science
Superior to “brainless” approaches
- “Ginny!” said Mr. Weasley, flabbergasted. “Haven’t I taught you anything? What have I always told you? Never trust anything that can think for itself if you can’t see where it keeps its brain?” from Harry Potter and the Chamber of Secrets, J.K. Rowling.

Use a mixture

Use a mixture of science/medicine with automated approaches?
- Story about an industrial process

At a minimum, use your knowledge as you evaluate these models. Avoid the tendency to think of a stepwise approach or any other automated approach as the final word. Think of it as an intelligent assistant, who might save you time, but you still have to carefully check everything they produce.

There’s an interesting anecdote that I heard many years ago (so long ago that I can’t remember the source). A researcher was examining an industrial process and was rating the importance of various factors in controlling the output. The statistician ranked all the variables using some sort of stepwise algorithm. They listed the most important variable, second most important, etc. and conclude with “and the least important variable is the amount of water present.”

At the point the whole audience erupted in laughter. They knew that water was very important because even a small amount of water in the system would cause a terrible explosion. In fact, the factory made every attempt to tightly control the amount of water. This control led to a very small standard deviation and very little power to detect a significant effect. The moral is to never blindly trust a model built solely on the data with no consideration of previous research and the underlying scientific and medical mechanisms.

Live demo, stepwise regression

Break #4

What you have learned
- Stepwise regression
What’s coming next
- Residual analysis

Using residuals to check for assumption violations

Non-normality
- QQ plot
Lack of independence
- Time sequence plot, Durbin-Watson statistic
  - Only for time-ordered data
Unequal variances
- Sactterplot of residuals versus predicted values
Non-linearity
- Scatterplot of residuals versus each independent variable
- Scatterplot of residuals versus predicted values

There are three important assumptions in multiple linear regression. You check the normality assumption with a Q-Q plot of the residuals. Nte that it is the residuals and not the dependent variable itself that you use here.

If your data has a natural ordering over time, then a time sequence plot and the Durbin-Watson statistic can help detect some violations of the independence assumption. You often do not have a time ordering for your data, and there may be other types of problems with independence associated with factors other than time.

The assumption of equal variances is sometimes violated when larger predicted values have greater variation. This occurs in quite a few setting.

Consider a mutliple linear regression model with housing prices as the dependent variable. You go to some older neighborhoods in the Kansas City area and the houses are mostly smaller and packed closely together on tiny lots. These houses might vary in price from 80 to 120 thousand dollars. Other neighborhoods have houses that are not so old, a bit larger, and a bit more space for your yard. These houses might sell across a broader range, maybe from 200 thousand to 400 thousand dollars. Then you have the neighborhoods with newer houses, lots of room both inside and outside. These houses might sell for 800 thousand dollars to 2 million dollars.

Notice how the variation marches in lock step with the prices. Small average prices have relatively little variation and large average prices have relatively large variation.

This also occurs with many other outcome variables that are bounded below by zero. The bound is quite constraining when you are on the low end, but less constraining on the high end.

The way to look for the pattern is to plot the residuals versus the predicted values. A fanning out pattern, with more variation with residuals associated with larger predicted values, is a violation of the assumption of equal variances.

Linearity is an implicit assumption of multiple linear regression. Sometimes, however, the relationship is nonlinear between one of your independent variables and your dependent or outcome variable variable. You might, for example, see an increasing relationship, but the increase tends to slow down at the high end. This is sort of like diminishing returns. Or perhaps the opposite occurs, when your independent variable is on the high end, the outcome takes off almost exponentially. Sometimes you can have too much of a good thing. A moderate value for your independent variable leads to a large outcome, but larger values backfire and bring your outcome back down to earth.

A plot of the residuals versus each independent variable is helpful here. A random scatter of points is a sign that the linearity holds. Any trend indicates that there is a more complex relationship between your independent variable and your outcome.

If you have more than a few independent variables, you can substitute a single plot the plot of residuals versus the predicted value. Think of the predicted value as a composite or linear combination of all your independent variables. If there is no obvious pattern or trend in this plot, then there is a good chance that you wouldn’t see any obvious pattern or trend in a plot of any of your independent variables and your residuals.

Q-Q plot of residuals

Scatterplot of residuals and chest circumference

Scatterplot of residuals and abdomen circumference

Scatterplot of residuals and hip circumference

Scatterplot of residuals and predicted values

Live demo, residual plots

Break #5

What you have learned
- Residual analysis
What’s coming next
- Collinearity

What is collinearity?

Strong interrelationship among the independent variables
- Also known as
  - multi-collinearity
  - near collinearity
  - ill-conditioning
Interrelationship could be just two variables
- Also could be three or more interrelated variables

Examples of collinearity

Birthweight and gestational age predicting length of stay
Size of the home and size of the lot predicting sales price
Calories from fat, from protein, and from carbohydrates predicting weight gain

If you have to predict length of stay in the birth hospital using birthweight and gestational age, there is a relationship between the two variables. Lower birthweights are associated with earlier gestational age and higher birthweights are associated with later gestational ages.

If you are trying to estimate what a house will sell for, the size of the house itself and the size of lot it sits on are both important. Larger houses tend to be found on larger lots and smaller houses tend to be found on smaller lots. There are some exceptions, of course, but the interrelationship between house size and lot size is strong.

A more complex relationship involves calories consumed from three sources: fat, protein, and carbohydrates. If you get a lot of calories from fat, there is less room in your diet for calories from protein and carbohydrates. You can say a similar thing for calories from protein or calories from carbohydrates. Now some people will just eat a lot of everything, but most people gravitate to one source of calories or another.

Now none of these interrelationships are perfect. Sometimes you will see a slightly underweight baby in a full term birth, for example. But there is a general relationship between the two in spite of a few exceptional babies.

What problems are caused by collinearity?

Difficulty in variable selection
Loss of precision
- wider confidence intervals
Loss of power
- Need for larger sample sizes
Not a violation of assumptions
Not a problem if you are only interested in prediction

When two or more independent variables are interrelated, it becomes difficult to decide where one variable or the other or both are needed to provide good predictions.

You will see a loss in precision. Your confidence intervals will be wider.

You also have less power. The sample size needed will have to be larger.

Think of a table with four legs. Normally you would put the legs in all four corners. This provides stability. But if the legs go along the diagonal fron one corner to the opposite corner, you don’t have good support in the remaining two corners. The table wobbles and is unstable.

Now the presence of multicollinearity does not make a multiple linear regression model invalid. You’re not violating any of the assumptions the you need. Instead think of collinearity as a data insufficiency problem. You have data on small babies with early gestational age, large babies with later gestational age, but almost no data on large babies with early gestational age and almost no data on small babies with late gestational age.

Fixing collinearity

Collect more data
Oversample “rare” corners
Prune your variables

Measures of collinearity

Correlation matrix
Tolerance
- \(Tol_i=1-R_i^2\)
  - \(R_i^2\) for predicting \(i^{th}\) independent variable from remaining independent variables
Variance inflation factor
- \(VIF_i=\frac{1}{Tol_i}\)
- Increase in \(Var(\hat\beta_i)\) due to collinearity

A simple way to check for multicollinearity is to examine the correlations among the independent variables. This will catch some of the simpler forms of multicollinearity, especially when it is an interrelationship among only two independent variables.

Tolerance is defined as 1 minus the R-squared value in a model predicting the \(i^{th}\) independent variable using the remaining independent variables. A small value for at least one tolerance is an indication of trouble with collinearity. If you can account for most of the variation in one independent variable using all the other independent variables, that implies a strong interrelationship.

How small is small? I would say 0.1 or less, but others have suggested the 0.4 or smaller is an indication of collinearity.

The variance inflation factor is a comparable statistic. It is just the reciprocal of tolerance.

The variance inflation factor is the reciprocal of tolerance. It has a direct interpretation. It represents how much larger the estimated variance is for an estimated slope in multiple linear regression due to collinearity compared to the ideal case where there would be no correlation between any of the independent variables.

I view a variance inflation factor of 10 or greater to be worthy of concern.

Collinearity statistics for the fat dataset

What is perfect collinearity?

Exact relationship among independent variables
Impossible to estimate regression coefficients
Examples
- Measuring temperature in both Fahrenheit and Centigrade
- Thee percentages adding up to exactly 100%
Only solution: drop one or more variables

Although most collinearities represent only approximate interrelationships, sometimes you might encounter a perfect collinearity. If there is a precise relationship rather than an approximate one, then you cannot estimate the regression coefficients at all.

Some examples of perfect collinearities are when you have the same measurement, just with different units, such as temperature in Fahrenheit and temperature in Centigrade. Or three variables representing percentages of a total might be perfectly collinear if they total up to exactly 100%. This is a perfect collinearity because once you know two of the percentages, you can estimate the third one precisely.

The only solution to a perfect collinearity is to drop one or more independent variables. Keep temperature in Fahrenheit or Centigrade but not both. Drop one of the three percentages that add up to exactly 100%.

Live demo, multicollinearity

Break #6

What you have learned
- Collinearity
What’s coming next
- Mediation

What is mediation?

“A situation when the relationship between a predictor variable and an outcome variable can be explained by their relationship to a third variable (the mediator)”
- Andy Field, Section 11.4

Why do you need to understand mediation?

Mostly irrelevant for prediction
Very important for understanding mechanisms
- Is there a direct relationship?

An informal assessment of mediation

There’s a relationship between age and percentage body fat. It is a bit easier to follow if you convert age in years to age in decades. The relationship between decades of age and body fat is that you add about 1.8% to your body fat every decade on average.

The question is, how does this happen? Does some of the muscle turn directly into fat, or do you add fat through an expanding girth?

When you look at a model with both decade of age and abdomen circumference, the effect of age is cut markedly. Instead of adding 1.8% body fat on average, you add about 0.7% body fat on average. The rest of the body fat comes from the “love handles” that you develop as you age.

So gaining some fat is inevitable, even if you can still fit into those size 32 jeans that you wore as a young man. But much of the gain in percentage body fat comes from moving from size 32 over time to size 42.

Live demo, mediation

Summary

What you have learned
- Three models for predicting a continuous outcome
- Analysis of variance table
- Variable selection
- Stepwise regression
- Residual analysis
- Collinearity
- Mediation

Additional topics??

Learning objectives
- List the differences and similarities between Simple Linear Regression and Multiple Regression
- Write the Null and Alternative Hypotheses for a Multiple Regression analysis
- Write the research question for a multiple regression analysis
- Identify the variable type for the Outcome Variable in a Multiple Regression analysis
- Identify the variable type(s) for the Predictor Variables in a Multiple Regression analysis
- Write the equation form for a Multiple Regression analysis
- Define Linear Combination
- Define the conditions for which Multiple Regression analysis would be applied
- Identify and detail what must be considered before running a Multiple Regression analysis
- Explain the Rule of 15
- Calculate the sample size for a Multiple Regression analysis
- List the assumptions that must be met for a Multiple Regression analysis to proceed
- Define: multicollinearity and homoscedasticity of residuals
- List the methods used to assess for all assumptions
- Explain why multicollinearity should never be present between the variables used in a Multiple Regressions Analysis
- List and explain the methods of variable entry into an Multiple Regression model
- List and Explain the methods of cross-validation of a Multiple Regression model
- Explain how predictor variables are selected for inclusion in a Multiple Regression analysis
- List all of the pertinent output found for a Multiple Regression analysis
- Define: Variance Inflation Factors and Tolerance
- Relate Variance Inflation Factors and Tolerance
- Define the interpretation of Variance Inflation Factors and Tolerance
- Explain the use of Cooks Influence
- List and define R, R squared and Adjusted R squared
- Explain the meaning of the ANOVA table generated in a Multiple Regression analysis
- Explain the difference between the standardized and unstandardized Beta Coefficients
- Interpret the Unstandardized Beta Coefficients from a Multiple Regression analysis
- Explain which predictors are included int he resultant Multiple Regression model
- Completely write up a Multiple Regression analysis